This paper presents Dream-RSI, a framework that lets AI agents improve their exploration strategies by learning from past discovery attempts without constantly running expensive new experiments. The system uses historical search data as a simulator to test and refine exploration policies cheaply before deploying them back into the real search space.

• Solves the exploration bottleneck in recursive self-improvement by creating a replay simulator from accumulated discovery history, enabling off-policy feedback without costly online re-evaluation
• Keeps the exploration layer separate from the underlying agent, making strategies explicit and programmable while maintaining compatibility with existing systems
• Demonstrates competitive or better results across algorithm engineering, mathematical optimization, and GPU kernel engineering while substantially cutting discovery costs